Skip to content

Publish wheels using older aarch64 images - #158

Closed
rhpvorderman wants to merge 10 commits into
mainfrom
patch1.4.0
Closed

rhpvorderman wants to merge 10 commits into
mainfrom
patch1.4.0

Conversation

@rhpvorderman

Copy link
Copy Markdown
Collaborator

Checklist

  • Pull request details were added to CHANGELOG.rst
  • Documentation was updated (if needed)

The most recent ones cause segmentation faults. This is a dummy PR to start the actions

@rhpvorderman
rhpvorderman changed the base branch from develop to main October 7, 2023 18:19
@tsibley

tsibley commented Nov 6, 2023

Copy link
Copy Markdown

The "Release x86 platforms only" commit here seems to be the reason why the recent 1.5.2 release of isal doesn't include aarch64 wheels. Since the 1.5.1 release did include those wheels, are there plans to restore them for 1.5.2? Or will they not be provided any more going forward?

@rhpvorderman

Copy link
Copy Markdown
Collaborator Author

@tsibley There is an issue with cibuildwheel on github actions. It somehow causes a segmentation fault when the wheel is tested.
I have tried my best to debug the issue (see #176 ) but I cannot find a reason why a segfault should be thrown. The address sanitizer finishes the test_zlib_compliance module without any hassle. For some reason the wheel crashes on test_zlib_compliance.

I am not going to provide faulty wheels, and I have already spent more time debugging the issue than I can justify. I don't use any aarch64 hardware at work. So if the wheels magically start passing tests again they will be published, if not I will let it be. If you are in the position to actually test why those wheels are corrupted you are welcome to make a PR to fix the issue.

@tsibley

tsibley commented Nov 8, 2023

Copy link
Copy Markdown

@rhpvorderman Thanks for your response. I appreciate not providing faulty wheels and your debugging efforts. It sounds like we should arrange for isal aarch64 builds ourselves in our system instead of relying on pre-built wheels. I'm not sure if we'll be in a position to figure out the issue with isal's CI builds, but if we are of course we'll contribute back.

The address sanitizer finishes the test_zlib_compliance module without any hassle. For some reason the wheel crashes on test_zlib_compliance.

Hmm. This is somewhat outside the area of my expertise, but it seems to me that libasan was not in effect for those tests? I see this in the workflow job logs:

ERROR: ld.so: object '/opt/rh/devtoolset-10/root/usr/lib/gcc/aarch64-redhat-linux/10/libasan.so' from LD_PRELOAD cannot be preloaded: ignored.

immediately before the second pytest invocation (the one which tries to enable libasan). That invocation also eventually segfaults:

/home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch-commands.sh: line 8:  4146 Aborted                 (core dumped) PYTHONDEVMODE=1 PYTHONMALLOC=malloc LD_PRELOAD=$(gcc -print-file-name=libasan.so) python3.8 -m pytest tests

@tsibley

tsibley commented Nov 8, 2023 •

Copy link
Copy Markdown

The libasan path being preloaded is not a shared library:

[root@ff75194faeae /]# file $(gcc -print-file-name=libasan.so)
/opt/rh/devtoolset-10/root/usr/lib/gcc/aarch64-redhat-linux/10/libasan.so: ASCII text
[root@ff75194faeae /]# cat $(gcc -print-file-name=libasan.so)
/* GNU ld script */
OUTPUT_FORMAT(elf64-littleaarch64)
INPUT ( /usr/lib64/libasan.so.6 )

Looks like some sort of "library by reference" meant to be processed by ld?

Ah, yes, from man 1 ld:

   If the linker cannot recognize the format of an object file, it
   will assume that it is a linker script.  A script specified in
   this way augments the main linker script used for the link
   (either the default linker script or the one specified by using
   -T).  This feature permits the linker to link against a file
   which appears to be an object or an archive, but actually merely
   defines some symbol values, or uses "INPUT" or "GROUP" to load
   other objects.  Specifying a script in this way merely augments
   the main linker script, with the extra commands placed after the
   main script; use the -T option to replace the default linker
   script entirely, but note the effect of the "INSERT" command.

@rhpvorderman

Copy link
Copy Markdown
Collaborator Author

immediately before the second pytest invocation (the one which tries to enable libasan). That invocation also eventually segfaults:

That segfault is the cause of the gzip compliance testing running python without PYTHONMALLOC=malloc, which causes the libasan crashes. This is a known issue (for me). I am primarily interested in test_zlib_compliance, which crashes when the wheels are built.

Hmm. This is somewhat outside the area of my expertise, but it seems to me that libasan was not in effect for those tests? I see this in the workflow job logs:

Strange. As later it does crash on libasan errors. That seems to me as it is working correctly. Regardless, I think the error is not in the python-isal code. I think it has something to do with the build process that cibuildwheel uses. I have no idea to fix it. At this moment I hope it just disappears again at some point.

@rhpvorderman

Copy link
Copy Markdown
Collaborator Author

Looks like the issue fixed itself! 1.5.3 is released with aarch64 images.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants